How can you detect and handle file encoding (e.g., UTF-8 vs UTF-16)?

Question

ICSM Computer · Answer

To detect and handle file encoding (e.g., UTF-8 vs UTF-16) in C#, you generally do the following:
1. Check for BOM (Byte Order Mark)
Many text files begin with a BOM that identifies the encoding. You can use 
StreamReader.CurrentEncoding after reading begins, or inspect the file header manually.
2. Use StreamReader with encoding detection
using System;
using System.IO;
using System.Text;
class Program
{
    static void Main()
    {
        string path = @"C:\Docs\example.txt";
        using (var fileStream = File.OpenRead(path))
        using (var reader = new StreamReader(fileStream, detectEncodingFromByteOrderMarks: true))
        {
            string content = reader.ReadToEnd();
            Encoding encoding = reader.CurrentEncoding;
            Console.WriteLine("Detected Encoding: " + encoding.EncodingName);
            Console.WriteLine("Content: " + content);
        }
    }
}
detectEncodingFromByteOrderMarks: true enables auto-detection of UTF-8, UTF-16 LE/BE (with BOM).
Limitations:
Only works reliably if the file includes a BOM.If no BOM is present, .NET defaults to UTF-8.
3. Manual BOM inspection for more control
Encoding DetectEncoding(string filePath)
{
    byte[] bom = new byte[4];
    using (var file = new FileStream(filePath, FileMode.Open, FileAccess.Read))
    {
        file.Read(bom, 0, 4);
    }
    // Check BOM
    if (bom[0] == 0x2b andand bom[1] == 0x2f andand bom[2] == 0x76) return Encoding.UTF7;
    if (bom[0] == 0xef andand bom[1] == 0xbb andand bom[2] == 0xbf) return Encoding.UTF8;
    if (bom[0] == 0xff andand bom[1] == 0xfe) return Encoding.Unicode;         // UTF-16 LE
    if (bom[0] == 0xfe andand bom[1] == 0xff) return Encoding.BigEndianUnicode; // UTF-16 BE
    if (bom[0] == 0 andand bom[1] == 0 andand bom[2] == 0xfe andand bom[3] == 0xff) return Encoding.UTF32;
    return Encoding.Default; // Fallback
}
Best Practices:
Write files with BOM if you expect them to be shared or read by unknown systems.Use UTF-8 without BOM if you're creating log or data files for internal use.For multi-lingual or international text, prefer Encoding.UTF8 or
Encoding.Unicode.

interview

How can you detect and handle file encoding (e.g., UTF-8 vs UTF-16)?

ICSM Computer

Can you answer this question?

1 Answers

1. Check for BOM (Byte Order Mark)

2. Use `StreamReader` with encoding detection

Limitations:

3. Manual BOM inspection for more control

Best Practices:

Liked By